Add Book to My BookshelfPurchase This Book Online

Chapter 8 - Troubleshooting

Cisco TCP/IP Routing Professional Reference
Chris Lewis
  Copyright © 1999 The McGraw-Hill Companies, Inc.

Troubleshooting Interface Problems
In this section, we'll examine serial, asynchronous, Ethernet, and Token-Ring interfaces, the most commonly implemented types of interface.
Troubleshooting Serial Interfaces
Serial interfaces are used with many different types of encapsulation: Cisco HDLC, PPP, frame relay, X.25, etc. In this section, we'll examine serial interface troubleshooting in some depth, assuming the default Cisco HDLC encapsulation. Most of what is done here is applicable no matter what the interface encapsulation is. Troubleshooting activities that are specific to one type of encapsulation will be examined in the section on troubleshooting protocol problems.
Show Interface Serial Revisited.     When connectivity problems arise on a specific serial interface, the first place to start is always with the show interface serial X command, to analyze its screen display. Figure 8-3 showed a typical screen display generated by the show interface serial command, and Table 8.1 explains each entry of interest to us in this display.
Table 8.1: Explanation of Show Interface Serial Command Display:
Screen Output
Explanation
Serial 0
Displays whether the interface is up, down, administratively down, or disabled.
line protocol
Displays an up down or looped condition.
Hardware is
Identifies the hardware type with a code.
MTU
Maximum Transmission Unit, the maximum message size that will be sent out this interface before it is fragmented.
Bandwidth
The value used by certain routing protocols (such as IGRP) for calculating a metric to associate with this interface's link.
DLY
The value for delay that the IOS assigns to a link of this bandwidth for metric calculation purposes.
Rely
The reliability of the link measured as a fraction of 255. This value is used in metric calculations for which it is usually taken to be 1.
Load
The load on the interface bandwidth, as a fraction of 255, which is used by some routing protocols for metric calculation.
Encapsulation
This reports which Data Link layer protocol is used for encapsulating packets sent on this interface.
loopback
Displays whether a software loopback condition has been set.
Keepalive
Indicates whether keep-alive packets have been explicitly defined for this interface.
last input, last output
Length of time since last packet input or output on this interface.
Last clearing of show interface
Reports when the last time the interface statistics, such as drops, were cleared.
output queue
Reports the size of the output queue and the number of packets dropped on this interface since the last clearing of statistics.
input queue
Reports the size of the input queue and number of packets dropped on this interface since the last clearing of statistics.
input rate
The rate of traffic inbound on this interface, measured as an average over 5 minutes and quoted in bits per second and packets per second.
Output rate
The rate of traffic inbound on this interface, measured as an average over 5 minutes and quoted in bits per second and packets per second.
Packets and byte input
The number of packets and bytes input since the last clearing of counters.
Received
A breakdown of the type of traffic received, as broadcasts, runts (packets shorter than allowed by the encapsulation method used), giants (long packets), and input errors such as CRC, frame, and overrun errors.
output
Number of packets and bytes output since the last clearing of counters, along with appropriate output error statistics.
interface resets
The number of times the line protocol has had to restart due to an interruption in end-to-end connectivity.
carrier transitions
The number of times the DCD signal has been lost and returned.
EIA signals
The status of DCD, RTS, CTS, DTR, and DSR.
Interface and Line Protocol Status in Detail.    Let's look at the possible combinations of interface and line protocol status that can be reported in the show interface serial command.
  Serial 0 up, line protocol up
  Serial 0 up, line protocol down
  Serial 0 down, line protocol down
  Serial 0 up, line protocol up (looped)
  Serial 0 up, line protocol down (disabled)
  Serial 0 administratively down, line protocol down
The Serial 0 interface up and the line protocol up is the fully working condition. Both the interface and line protocol have initialized and protocol keep-alives are being exchanged.
The most common fault condition, Serial 0 up but line protocol down, can be due to a variety of conditions. This display tells you that the router is connected to a device that is providing a DCD signal, indicating that a carrier signal is present between the local and remote CSU/DSU. Due to a problem with router configuration or CSU/DSU operation, however, protocol keep-alives are not being exchanged properly between the two ends of the connection. This condition is reported if there are problems with clocking on the CSU/DSU, if the two serial interfaces connected via the link are not on the same subnet, when there is a noisy leased line or a remote router failure of some kind.
The best way to troubleshoot this condition is to follow the loopback tests outlined in a later section in this chapter. If these procedures do not identify the problem, and you are sure both routers are configured correctly, use the debug serial interface command (also covered in a later section in this chapter) to identify clocking problems. If all else fails, start to replace pieces of hardware until the faulty device has been located.
If both the Serial 0 interface and the line protocol are down, it means that the serial interface is not sensing a carrier detect (DCD) signal from its attached CSU/DSU. This can be due to a telephone company line problem, a cabling issue or a CSU/DSU failure. The most frequent cause of this display is a problem with the telephone company's leased line. Following loopback tests, outlined later in this chapter, will identify which link in the chain is causing the problem.
When you put a CSU/DSU in loopback mode, the condition of Serial 0 up and the line protocol up (looped) is displayed. The router interface recognizes that it is receiving the same keep-alive messages it is sending. If you see the condition reported when the local CSU/DSU is in loopback, you know all is good with the router and CSU/DSU interface and connecting cable. If this condition is displayed when just the remote CSU/DSU is in loopback, you know all is good up to the line interface of the remote CSU/DSU. If the line protocol will not come up when the loopback is removed from the remote CSU/DSU, there is something wrong with the remote CSU/DSU DTE interface, the cable between the remote CSU/DSU and the remote router, or the remote router itself (including its configuration).
The condition of Serial 0 up and line protocol down (disabled) is seen if the router interface has received more than 5000 errors during one keep-alive period (which is 10 seconds by default). The most likely cause of this condition is either a faulty CSU/DSU device at either end of the link, or a faulty telephone company line.
Finally, the condition of Serial 0 administratively down and line protocol down is reported when the shutdown command has been entered into the configuration of the interface. The condition is removed by taking the entry out of the interface configuration by entering the no shutdown command.
Troubleshooting Packet Drops and Errors.     Once we have an up condition for both the interface and line protocol, the basic communications across a serial link are established. There are, however, plenty of potential problems that can arise, and in this section we'll examine problems reported with packet drops and errors.
The first place to look is the status of the input and output hold queues in the show interface serial command output. Hold queues are specific to an interface and are set while in interface configuration mode. If you see that the number of packets in the hold queue (the hold queue defines the number of packets that can be held awaiting hand-off to a transmit buffer) is reaching the maximum allowed, it is worth trying to increase the size of the hold queue. It is possible that by doing this, more misses appear at the stage at which packets are buffered as reported by the show buffers command. We will examine router buffers and how they can be optimized later in this section.
The way to think about a packet passing through a router is that it comes into an interface hold queue, gets passed off to a system buffer of an appropriate size, and is switched to the desired interface to be sent to its next hop destination.
The nature of LAN traffic means that it is bursty, i.e., lots of packets arrive at one time instead of in a constant predictable stream. This makes it difficult to issue the show interface serial command at the right moment and catch overutilization of hold queues. The show serial interface command is a static display, a snapshot at the moment you issue the command. If you see drops reported after the input or output queue usage statistics, however, it costs little to increase the size of these queues and see if that improves matters. An example of increasing an output hold queue from its default of 40 to 100 packets is given as follows:
Router1(config)#interface serial 0
Router1(config-int)#hold-queue 100 out
The other configuration change that should be made as a matter of course when experiencing packet drops on an interface is for the no ip route-cache command to be entered in to the configuration of the interface experiencing drops. This command forces packets routed through this interface to be process switched rather than fast switched. Fast switching means that packet destinations are held in memory and the routing decision is made without interrupting the route processor. Route switching means that the route processor will be involved in each packet routing decision. The net effect of using this command to make packets process switched is that it slows down traffic passing through the interface. At first this may seem against your design goals, but when you are switching packets from an ethernet to a 64K or even T-1 line, it is better to slow down packets through an interface than to have them dropped because an interface is overwhelmed with packets.
The next place to look for problems if packets are being dropped is the show buffers command, a sample output of which is illustrated in Fig. 8-5.
router3#show buffers
Buffer elements:
406 in free list (500 max allowed)
114 hits, 0 misses, 0 created
Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50):
50 in free list (20 min, 150 max allowed)
5 hits, 0 misses, 0 trims, 0 created
Middle buffers, 600 bytes (total 25, permanent 25):
25 in free list (1 0 min, 150 max allowed)
12 hits, 0 misses, 0 trims, 0 created
Big buffers, 1524 bytes (total 50, permanent 50):
50 in free list (5 min, 150 max allowed)
5 hits, 0 misses, 0 trims, 0 created
VeryBig buffers, 4520 bytes (total 10, permanent 10):
10 in free list (O min, 100 max allowed)
10 in free list (O min, 1 00 max allowed)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 0 trims, 0 created
Interface buffer pools:
Ethernet0 buffers, 1524 bytes (total 32, permanent 32):
8 in free list (0 min, 32 max allowed)
24 hits, 0 fallbacks
8 max cache size, 8 in cache
Serial0 buffers, 1524 bytes (total 32, permanent 32):
7 in free list (0 min, 32 max allowed)
25 hits, 0 fallbacks
8 max cache size, 8 in cache
Serial1 buffers, 1524 bytes (total 32, permanent 32):
7 in free list (0 min, 32 max allowed)
25 hits, 0 fallbacks
8 max cache size, 8 in cache
0 failures (0 no memory)
Figure 8-5: Output of the show buffers command
A router has buffers of five different sizes, and will place an incoming packet in the smallest buffer size possible. With reference to Fig. 8-5, you can see that on an Ethernet network with a maximum packet size of just over 1500 bytes, no "verybig," large, or huge buffers will be utilized. On networks that use Token-Ring, on which the maximum packet size is more than 4 kilobytes, some verybig or large buffers may be used. Let's examine what each entry in this display means.
Buffer Elements.     These are nothing that you should be directly concerned with; they are markers used by the IOS to keep track of buffers and where they reside in memory.
Hits, Misses, Trims, and Created.     A hit occurs when an attempt to allocate a buffer is successful, and indicates the desired state of affairs regarding the operation of available buffer resources. A miss is not necessarily a bad thing, although its name can give rise to some concern. The number of misses is the number of times the buffer pool needed to grow because a buffer was required, but no buffer of the appropriate size was available. Trims and created reflect the number of buffers deleted and added for a given buffer size and illustrate the dynamic nature of system buffer allocation in Cisco routers.
Small, Middle, Big, Verybig, Large, and Huge Buffers.     These names represent blocks of main memory used to hold network packets.
Total, Permanent, Free List, Min, and Max Allowed.     These values are quoted for each size of buffer. The total number of buffers of this size is quoted, along with an indication of how many of these are permanent and will not be trimmed by the operating system if they are unused. The min and max allowed display the minimum number and maximum number of this size of packet that the operating system will allow to be free at any given time.
Interface Buffer Pools, Fallbacks, and Failures.     The interface buffer pools are small amounts of fixed memory allocated to packets passing through an interface. These buffers cannot be directly manipulated by router configuration commands. The display for the interface buffer pool contains many entries that are the same as for the public buffer pools. The exceptions are fallbacks and failures. Fallbacks occur when a buffer cannot be allocated from the interface pool and needs to be allocated from the public pool, which is a normal occurrence and does not warrant any particular action. Failures are a bad thing, since the failures are the number of times a buffer allocation failed because a buffer was not available, which results in a lost packet. The display additionally tells you how many of these failures were a result of no memory being available to create a new buffer.
Once you have optimized hold queues and buffers as much as possible, and potentially increased the amount of physical memory available for main RAM, the only option left for reducing persistent drops is to increase the bandwidth on the link.
In practice, trial and error is generally the only way to optimize the size of hold queue and buffer pool. The statistic with which you should be most concerned is the number of drops reported by the show interface command. If you are experiencing drops, first try increasing the size of the hold queue. If this does not help, it might be that there are not enough buffers of a certain size. The system will create more buffers as they are needed, but this takes time.
While the system is creating more buffers, it is possible for the hold queue to be overrun and to start dropping packets. If you suspect this might be the case, you will need to increase the number of buffers of the size that is reporting misses. A similar situation can occur if you see a lot of trims and created for a given size of buffer, since it takes time for the router to add or delete a buffer. If you see lots of trims or created, you should increase the max-free and min-free amounts for the buffer size in question. The effect of increasing the min-free and max-free value should ensure that there always will be enough buffers available to take incoming packets, without the need for the system to create more buffers.
In addition to reporting drops experienced on a given interface, the show interface serial command also gives useful information on many of the different types of errors that can be encountered on a line. The errors are grouped under input and output errors as follows:
  Input errors: CRC, frame, overrun, abort
  Output errors: collisions, interface resets,  carrier transitions
Input errors generally come from a faulty leased line, CSU/DSU hardware, or out-of-spec cable. CRC errors mean that the packet received is not the same as the packet sent, indicating that there is interference on the transmission path. All cables, telephone company lines, and CSU/DSU services should be checked if this value is 1 percent or more of total traffic.
A framing error means that the packet received does not end on an 8-bit byte boundary. Errors of this type can come from interference of the kind that results in a CRC error. Additionally, framing errors can occur whenever CSU/DSU setups (framing and line coding values) on either end of the link are incompatible with the framing of the line in use. All internetwork devices should be configured to use a clock from one source, because if devices on the one link are using disparate clock sources, framing errors can occur.
Overruns are rare; they occur if packets arrive too quickly for the router to handle. This typically is a problem only if the router CPU utilization is very high and all you can do is upgrade to a router with a faster processor, or try to offload some of the work of the router to another device.
Abort packets occur when an illegal sequence of bits has been received. Typically, to maintain synchronization, a sequence of seven or more 1s is not allowed by line coding schemes. If seven or more 1s appear in a data stream, 0s are inserted to transmit the packet over the line, and are removed at the receiving end. Therefore, a packet containing seven 1s is a code violation and results in an abort.
Output errors generally have a lesser impact on internetwork performance than input errors. The show interface serial command lists collisions, which are only really applicable to Ethernet media, so rarely will this number be anything other than 0.
If interface reset errors are reported, it is because keep-alive packets are being missed. This could be caused by constant carrier transitions (which will be covered next), or significant output drops, or increasing numbers of input errors. Generally, one of these other reported problems must be resolved to resolve interface reset problems.
Carrier transitions are counted whenever the carrier signal changes state. This typically is due to a faulty line, or, on a less regular basis, CSU/DSU failure or faulty cabling.
Resolving Problems with Clock Sources.     A single reliable clock source is essential for synchronous communications over a telephone company data line. The speed of the clock source determines the bandwidth of the line that is available for data transmission. For example, a 128 kbps line will have a clock source running at 128,000 cycles per second. This means that the clock source will "clock" 128 kilobits of data onto the line in 1 second.
A string of data being transmitted down a line consists of 1s and 0s, represented by electrical signals at a given voltage level. The clock tells all devices that are communicating what length of time is used to represent a 1 or a 0. Let's say for argument's sake that the clock states that a constant voltage level for 8 ms represents 1 bit (roughly what our 128 kbps clock does). If a constant voltage level is transmitted for 16 ms, that means that 2 bits of the same value have been sent. You can see that if the communicating devices do not agree precisely on the clock signal, they will not be able to interpret data sent between them properly.
When you look at the connectors between a CSU/DSU and a router interface, you see many pins, some of which are dedicated to producing a clock signal. Between CSU/DSU devices, there is a telephone company line of some type that has far fewer connectors, typically two or four wires. Due to the lack of individual wires on a line, the clock signal is embedded in the data stream sent over the line. This means that the CSU/DSU must derive a clock signal from the data that passes through it.
To achieve this, the CSU/DSU must receive at least one 1-bit value every 8-bit byte. If the data stream contains eight or more 0s, the CSU/DSU encoding mechanism must insert a 1-bit value for transmission over the line and remove it from the data stream at the other end. This technique is referred to as maintaining one's density. The line supplied to you by the telephone company and your CSU/DSU must agree on the method of maintaining one's density, or the clock source will be lost and errors will occur. For example, T-1 lines now use Extended Superframe Format (ESF) framing with Binary 8-Zero Substitution (B8ZS). Previously, T-1 lines used Superframe Format framing with Alternate Mark Inversion (AMI) encoding.
If there are significant input errors (between 0.5 percent and 1.5 percent of traffic), you should check that all clock sources are set to the same source and that all use the same framing and encoding schemes.
So far we have talked about the clock source that is used to transmit data between CSU/DSU devices. A clock also is used to transmit data between the CSU/DSU and the router interface. The CSU/DSU retrieves a clock signal from the data it receives over the line to use for transmitting data to the router interface. On this link, no encoding schemes are necessary, as a separate wire carries the clock signal. When it comes time for the router interface to send data back to the CSU/DSU, the router generates its own clock and transmits it on a pin known as the serial clock transmit external (SCTE) pin. When the CSU/DSU device uses this clock rather than the one it derived over the line from data stream received, it is better able to decipher the data from the router without error. You should check that your CSU/DSU devices support and are configured for SCTE timing, especially if data rates above 64 kbps are used.
If an interface reports increasing input errors, leading you to suspect problems with the clock source, the best means of tracking down the problem are by using loopback and ping tests to identify which link in the chain from source to destination is causing a problem; these tests are covered in the next section. The following points summarize what we have discussed so far regarding the resolution of clocking problems:
  1. Check that both CSU/DSU devices take their clock source from the line.
  2. Check that the CSU/DSU device has a configuration that matches the line supplied by the telephone company; this includes framing, encoding, and other line characteristics, such as impedance level.
  3. Check that SCTE is in use by both CSU/DSU devices.
  4. Check that cable lengths are within specification and are of the shielded type if problems with interference on cables persist.
CSU/DSU Loopback Tests.     Loopback tests are useful for identifying where in the chain of links from source to destination either communications are failing, or errors are being introduced. Most CSU/DSU devices have an option to go into what is referred to as digital loopback. Digital loopback provides two loops, one from the CSU/DSU to the router interface and the other from the CSU/DSU to the telco leased line. By putting first the local CSU/DSU in loopback, as shown in Fig. 8-6, we can see if the connection from router 1 to CSU/DSU1 is okay by looking at the show interface serial 0 output on router 1. We also can view the state of the connection from router 2 all the way through the telco network to CSU/DSU1.
Figure 8-6: Putting a CSU/DSU in digital loopback
I have stated that loopback tests do not work for synchronous PPP connections, and neither do they work for X.25 or frame relay connections. If, however, the type of communications you're using allows loopbacks (such as the Cisco HDLC encapsulation), they are a useful troubleshooting tool. On more complex internetworks, where connections pass through many router devices, ping tests help you identify where connectivity fails and, potentially, which link is introducing errors in communication. To check connectivity, the ping command issued in nonprivileged mode sends 5 ICMP ping packets, each of 100 bytes. This is good for checking physical connectivity and ensuring that routing tables have all the necessary entries. To use the ping command to check the integrity of connections, you must be in privileged mode, which gives you options to set the size, number, and data pattern of ping packets sent. A typical screen display for pinging in privileged mode is given in Fig. 8-7.
router2#ping
Protocol [ip]:
Target IP address: 164.7.1.67
Repeat count [5]: 100
Datagram size [100]: 1000
Timeout in seconds [2]:
Extended commands [n]: y
Source address: 164.7.1.97
Type of service [0]:
Set DF bit in IP header? [no]:
Data pattern [0xABCD]: 0xFFF
Loose, Strict, Record, Timestamp, Verbose[V]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 1000-byte ICMP Echos to 164.7.1.67, timeout is 2 seconds:
Packet has data pattern 0x0FFF
Reply to request 0 (8 ms)
Reply to request 1 (4 ms)
Reply to request 2 (4 ms)
Reply to request 3 (8 ms)
Reply to request 4 (4 ms)
Reply to request 5 (4 ms)
Reply to request 6 (8 ms)
Reply to request 7 (4 ms)
Reply to request 8 (4 ms)
Reply to request 9 (4 ms)
Reply to request 10 (4 ms)
Figure 8-7: Use of ping in privileged mode
The extra functionality of the extended ping allows you to generate traffic on a serial link in order to test how the link operates under full-load conditions. Also, by specifying an all 1s or all 0s data pattern, the coding and framing configurations can be tested.
There are options to configure loopbacks for certain Cisco interface types. Essentially, Cisco interfaces that have a built-in CSU/DSU, like the 2524, 2525 or interfaces like the HSSI or channelized T-1 controller, will be able to use this command. However, the following loopback commands will have no effect if an X.21 interface is in use, as the X.21 specification does not include a loopback definition.
Loopback tests can be used on Ethernet, and token ring interfaces; however, it is most useful for serial interfaces, where the serial interface itself is the DTE and the associated CSU/DSU is the DCE.
An example of the use of this command is with the HSSI, that supports loopback dte, loopback line and loopback remote commands. The loopback dte command loops packets to the DTE within the CSU/DSU. This is useful in testing whether there are any problems with the interface itself. If this test passes, the loopback line command can be used, which loops packets through the CSU/DSU and thus tests the DCE device. The loopback remote places the CSU/DSU at the other end of the land line in loopback mode and loops packets across the land line and back again.
Debug Serial Interface Command.     There are many debug commands that can be used when troubleshooting serial communications, which are specific to the encapsulation used on the interface. We will examine some of these encapsulation-specific debug displays when we consider troubleshooting the specific protocol in question. At this stage, we will consider the debug serial command screen output generated when the interface is using the default Cisco HDLC encapsulation.
Before we look at this display, let's review how best to use debug mode on Cisco routers.
Debug mode potentially can use a lot of processor power and cause the router to drop a high percentage of packets. In fact, one sure way to make a router stop functioning until it is powered off, then on, is to issue the debug all command. This will debug all packets passing through the router and, if there is even a modest amount of network traffic, will bring the router to its knees.
The most efficient way to have the router report debug information is to use the logging buffered global command to have the debug command output written to a log file. The output then can be viewed with the show log command. At times you may elect to see debug messages appear on the terminal display as they are generated. This can be achieved either by attaching a terminal directly to the console port of the router, or, if accessing the router via Telnet, issuing the term mon command. This command copies the output going to the console to your Telnet session.
Debug information usually is checked once all other standard troubleshooting procedures have failed. If the line protocol continually reports a down condition when all configurations and connections appear to be good, you can use the debug serial interface command to see if a timing issue at either end of the communication link is interfering with the exchange of keep-alive messages. A typical screen output for the debug serial interface command is illustrated in Fig. 8-8.
Serial0: HDLC myseq 129, mineseen 129, yourseen 129, line up
Serial0: HDLC myseq 130, mineseen 130, yourseen 130, line up
Serial0: HDLC myseq 131, mineseen 131, yourseen 131, line up
Serial0: HDLC myseq 132, mineseen 132, yourseen 132, line up
Serial0: HDLC myseq 133, mineseen 133, yourseen 133, line up
Serial0: HDLC myseq 134, mineseen 134, yourseen 134, line up
Serial0: HDLC myseq 135, mineseen 135, yourseen 135, line up
Serial0: HDLC myseq 136, mineseen 136, yourseen 136, line up
Serial0: HDLC myseq 137, mineseen 137, yourseen 137, line up
Serial0: HDLC myseq 138, mineseen 138, yourseen 138, line up
Serial0: HDLC myseq 139, mineseen 139, yourseen 139, line up
Figure 8-8: Log output for the debug serial interface command
The myseq (my sequence), mineseen, and yourseen are the interesting parts of this display. During normal operation, the myseq and mineseen values are equal and increment after each keep-alive is sent by this router. The myseq is the sequence number of the keep-alive sent, and the mineseen is the sequence number last acknowledged by the remote router. The yourseen value is the last sequence number received from the remote router. It is normal for a keep-alive message to be lost once in a while, but three consecutive missed keep-alives causes the link to be reset. When the link is reset, the sequence numbers revert back to the last acknowledged sequence number values.
If everything else on the link looks good, but you are consistently missing keep-alives, you should report this condition to the telephone company that supplies your lines.
Troubleshooting Asynchronous Communications
Asynchronous communications are one of the most problematic areas in providing comprehensive internetwork services to an organization. We covered the security aspects of dial-up networking in Chap. 7 with CHAP and use of a TACACS server. The main challenge of asynchronous communications, however, is providing reliable service to users. The chief issue is minimizing drops while transporting what is typically LAN-based traffic over an asynchronous link. A LAN operates at 10 Mbps or above, whereas modems at best connect at 28.8 kbps, but very often, due to line conditions, only at 26.4 or 21.6 kbps. With V.42bis compression, throughput rates can exceed 50 kbps temporarily, but this can pose problems for many applications.
Assuming that you have minimized all service and route advertisements with the use of access lists for the protocols in use over the asynchronous link, the discussion in the previous section on dealing with input and output drops via manipulation of the hold queues and buffer pools applies. Exact settings for hold queues and buffers can be determined only on a case-by-case basis and may require the installation of more main RAM in the router.
Before we examine router-based troubleshooting of asynchronous communications, let's review external factors that can contribute to poor or nonperformance of a dial-up link. The first is the configuration of the PC hardware at the remote end. A remote PC must be using a 16550 UART (Universal Asynchronous Receive Transmit) chip in the circuitry of the serial port used to connect to the remote modem. Previous UART chips were inadequate; for example, the 8250 operated only at up to 9600 kbps and, although the 16450 operated at higher speeds, its buffer capabilities were too limited for it to be useful. If the remote PC does not use a 16550, throughput will be severely limited, which will increase the number of packets dropped by the router.
The next external factor is the quality of the connection made between the local and remote modem by the telephone company switch. Even in a perfect system, you should expect around 3 percent of connections to fail because of poor connections provided by the telephone company switch. Significantly more failures can be experienced if one of the local loops from the telephone company central office to either the local or remote modem is of poor quality.
Some higher-end modems provide a front panel display that reports the dBm receive and transmit levels, the signal quality, and the error rate of the attached line. A poor signal quality or high error rate means you should contact the telephone company and have them check that local loop. In many cases, telephone companies will check only to determine that a dial tone is delivered by the local loop and will look no further. If this happens, it is useful to be able to quote the receive and transmit dBm levels. The receive dbm level should be around-9 to -15 dBm; if it is out of that range, the telephone company should be able to get the line back in specification.
We have appropriate router access lists, hold queues, buffer settings, V.34 modems for both the router interface and remote PC (which uses a 16550 UART), and good local loop dial connections at both ends. What else do we need to check?
The next thing to check is the setup of the modems used at both ends of the link. The setup for the modem attached to the remote PC is not so much of a problem, because it normally is initiated every time you make a new dial connection. Most PC communication programs have you identify the modem you are using and send an appropriate setup string prior to issuing the dial command. The modem connected to the router interface is, however, a different story. That modem normally sits there waiting for a connection to arrive from a remote modem and does not receive a setup string prior to that connection being made. This means that the modem should receive its setup prior to it having to answer any calls, and should have that setup saved to memory. It also means that you should have a way to initiate a modem remotely, if one loses its configuration or needs to be replaced.
A simple way to do this is to use modems that allow configuration via a front panel. An alternative is to connect the modem to a router asynchronous interface and initiate a reverse Telnet session to the interface to send setup commands to the modem. Let's say you are logged on to router 1 and want to initiate a reverse Telnet session to the first asynchronous port. You would accomplish this by entering the following:
Router1>telnet 193.1.1.1 2001
This assumes that 193.1.1.1 is an interface on the router that has its line protocol up. The number 20 has to precede the number of the asynchronous interface to which you wish to connect. If router 1 is an access server, such as the 2511 (which has 16 asynchronous ports), the 01 interface will be the first asynchronous interface. If router 1 is a 2501, the 01 interface is the auxiliary interface. On a 2511, the auxiliary interface is number 17. A Telnet session to an asynchronous interface will be refused only if a user already is connected on the interface, or the modem inout line command has not been executed. (We will review the configurations necessary for the line section later in this section.)
Once the Telnet session has been established, you can verify connectivity by typing the letters AT followed by the Enter key, to get an OK echoed back from the modem. You then can send the setup string appropriate for your modem and set it up the way you want. A typical setup string will have the modem answer after one ring, lock in the DTE rate to a given value, hang up the connection if the DTR signal is not present, set the DCD signal to high only on carrier detect, enable compression, utilize hardware flow control, and write this configuration to memory.
The item in the modem configuration just given that needs further explanation is locking of the DTE rate, which is the speed of communications from the modem to the router interface. Generally, modems set the DTE rate to automatic, meaning that the modem will set its DTE rate according to the speed of the communication sent from the device connected to it. In the case of a modem connected to a router interface used for dial-in connections, this does not work. For a dial-in connection, the modem initiates communication with the router interface. This means that nothing is being sent from the router to the modem to set the modem DTE rate. Therefore, the modem DTE rate must be preset in the modem to equal the DTE rate of the router async interface for the modem to initiate communications. If you are using V.34 modems, setting the router interface and modem DTE rate for 115,200 bps will give you the best potential for the highest throughput.
For serial interfaces, the show commands we made use of were the show interface and show controllers commands. For asynchronous interfaces, the show interface and show line commands provide the most useful information.
Taking the first asynchronous interface as an example, the show interface async 1 command produces a screen display similar to that illustrated in Fig. 8-9.
router2#show int async 1
Async1 is up, line protocol is down
Hardware is Async Serial
Interface is unnumbered. Using address of Ethernet0 (164.7.1.66)
MTU 1500 bytes, BW 9 Kbit, DLY 100000 usec, rely 255/255, load 1/255
Encapsulation PPP, loopback not set, keepalive set (10 sec)
DTR is pulsed for 5 seconds on reset
lcp state = REQSENT
ncp ccp state = NOT NEGOTIATED ncp ipcp state = NOT NEGOTIATED
ncp osicp state NOT NEGOTIATED ncp ipxcp state = NOT NEGOTIATED
ncp xnscp state = NOT NEGOTIATED ncp vinescp state = NOT NEGOTIATED
ncp deccp state = NOT NEGOTIATED ncp bridgecp state = NOT NEGOTIATED
ncp atalkcp state = NOT NEGOTIATED ncp lex state = NOT NEGOTIATED
ncp cdp state = NOT NEGOTIATED
Last input never, oldput 0:00:00, output hang never
Last clearing of "show interface" counters never
Output queue 0/100, 0 drops; input queue 1/100, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
36 packets output, 864 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets, 0 restarts
0 output buff er failures, 0 output buffers swapped out
0 carrier transitions
Figure 8-9: Output of show interface async 1 command
The status reported for the interface still is given as up, even if carrier detect is not high, which is unlike the show serial interface display. The only time the interface will report a down condition is if the encapsulation type is not set for the interface. The line protocol display behaves the same as the line protocol for the serial interface. Other than the interface and line protocol status, the show interface async 1 command is as useful for monitoring throughput, drops, and resets as it is for serial interfaces.
One additional set of information that the show interface command provides is related to the fact it uses PPP encapsulation. With this configuration, the show command reports the status of the PPP, LCP, and NCP protocols. These reports are the most valuable when multiple protocols are being used on the one link. For example, if IP and IPX are being used on this interface, you would expect to see the ipcp and ipxcp state as negotiated when a user has dialed in. If the user is experiencing difficulty in accessing some applications and not others over the dial link, it might be because one of the protocols has not successfully negotiated. In this situation, the interface and line protocol will report up, and the only indication that one of the protocols in use over the link has failed is the state of the NCP negotiations.
The major omission for the show interface async 1 display is in reporting the status of EIA signals. This is given in the show line 1 command, which is illustrated in Fig. 8-10.
router2#show line 1
Tty TypTxtRxA Modem Roty AccO Acci Uses Noise Overruns
A 1 TTY 9600/9600 - - - - - 0   0   0/0
Line 1, Location: "", Type: ""
Length: 24 lines, Width: 80 columns
Baud rate (TX/RX) is 9600/9600, no parity, 2 stopbits, 8 databits
Status: Ready, Active, Async interface active
Capabilities: Line is permanent async interface
Modem state: Ready
Special Chars: Escape Hold Stop Start Disconnect Activation
^^X none - -  none
Timeouts: Idle EXEC Idle Session Modem Answer Session Dispatch
0:10:00 nevernonenot set
Session limit is not set.
Time since activation: never
Editing is enabled.
History is enabled, history size is 10.
Full user help is disabled
Allowed transports are pad telnet riogin. Preferred is telnet.
No output characters are padded
No special data dispatching characters
Modem hardware state: noCTS noDSR DTR RTS
Line is running PPP routing.
0 output packets queued, 1 input packets.
Async Escape map is 11111111111111111111111111111111
Figure 8-10: Output of show line command
The show line command displays the receive and transmit speeds set for that interface, whether hardware flow control has been set, and the status of the EIA signals CTS, DSR, DTR, and RTS. Unlike serial interfaces, in which all configuration is entered under the serial interface section, asynchronous interfaces have some things (such as encapsulation and hold queues) set on the interface, and DTE rate and modem control set under the line configuration. A typical line configuration that would appear in a router configuration file and be applied to asynchronous interfaces 1 through 16 is given as follows.
Line 1 16
modem inout
rxspeed 115200
txspeed 115200
All configuration changes made on line 1 will apply to asynchronous interface 1, and it is easiest just to think of the line commands as extensions to the interface configuration commands. The rxspeed and txspeed commands set the receive and transmit DTE rates to 115,200 (which has to match the setting for the DTE rate on the attached modem), and the modem inout command enables modem control for the interface.
The reporting of the EIA signals on the show line 1 display is interesting. If no modem is connected to asynchronous interface 1, the EIA signals report:
noCTS noDSR DTR RTS
If a modem is attached that has CTS high, the EIA signals are reported as:
CTS noDSR DTR RTS
which is the normal display for an interface connected to a modem on which no remote modems currently are connected. (Your modem should be configured to have CTS high by default, lowering the signal only to throttle traffic.) When a remote modem calls in and establishes a connection, carrier detect will be raised on the modem and the EIA signals will be reported as:
CTS DSR DTR RTS
The connection from a 2511 router to a modem on an asynchronous interface is via an RJ-45 cable, which has 8 wires. The function of these wires are assigned to RTS (request to send), DTR (data terminal ready), Txd (transmit data), Sig Gnd ( signal ground), Cab Gnd (cable ground), Rxd (receive data), DSR (data set ready) and CTS (clear to send). As we just saw, using the IOS show line command, the status of only four of these signals is reported. As long as you realize that the status of DSR reported by the command is more usefully interpreted as the status of the DCD signal, you can get by with reporting on just these four signals.
Debug Commands for Asynchronous Connections.     The most usual encapsulation for an asynchronous interface is the PPP encapsulation. If you're troubleshooting the interface and line configuration on the router and the modem connections, setup, and telephone line do not resolve problems, you can get additional information from the debug ppp commands.
The debug ppp command can be followed by one of four keywords: packet, negotiation, errors, or chap. The screen outputs from these commands are hard to understand and it's difficult to derive useful information from them. If this command does generate screen output, however, at least you know that everything on the Physical layer is working. If you have to resort to using these commands, start with the debug ppp errors and report these to Cisco technical support. The type of problems that are resolved by the debug commands are usually related to improper PPP commands being sent by a faulty PPP driver on the remote PC, which these days is a rare occurrence.
Asynchronous Troubleshooting Summary.     If you have a correctly configured remote PC and modem that can successfully dial in to one interface, but not to another, the steps outlined next will help identify where the problem lies.
  1. Use the show line command and check that the CTS signal is present, to prove that the modem is connected to the correct interface and powered up.
  2. Check that the line rxspeed and txspeed match that of the modem.
  3. Check all the modem settings and the operation of the telephone line attached to the modem. (Connecting the line to an analog telephone handset lets you hear if the line is okay.)
  4. Check encapsulation and CHAP setting for the interface.
  5. Use the debug ppp command to see if PPP errors are being reported and seek help from Cisco technical support if necessary.
If communication over the asynchronous link is poor, do the following:
  1. Check the number of drops and buffer pool misses and increase the size of hold queues or buffer pools that are experiencing problems.
  2. Check your modem for reports on the telephone line quality and dBm levels.
  3. Check the speed of modem connection; it should be at or above 21.6 kbps. If it is not, try to change telephone lines on either or both ends.
  4. Make sure that the DTE rate for all device interfaces is 115,200 bps and compression is enabled on both modems.
  5. Check that the remote PC has a 16550 UART chip in use in its serial port.
Troubleshooting Ethernet
Compared to the problems encountered with serial and asynchronous interfaces, troubleshooting the setup of Cisco Ethernet interfaces is simple. The most common problems on Ethernet interfaces have nothing to do with how the router is set up; rather, the problems typically involve overutilization of bandwidth, a high number of collisions, or systems using incompatible frame types.
Collisions, drops, utilization (as reported by throughput), and frame type are reported on the show interface ethernet 0 command as illustrated in Fig. 8-2. Other problems, such as LAN card drivers incorrectly forming frames that are either too long (giants) or too short (runts), are rare these days. The major challenges of implementing Ethernet networks are in the areas of physical design and management of cabling systems. We will review how Cisco IOS screen displays report typical Ethernet problems.
Once the Ethernet interface and line protocol report an up condition, which is achieved by connecting the Ethernet interface to a hub, little optimization can be achieved through interface configuration. The most significant factor in serial line performance problems we examined was with packet drops. You rarely see drops on an Ethernet interface, even on a heavily utilized segment. The only time drops are likely to become a factor is when you have a router connecting a heavily utilized FDDI ring and an Ethernet segment. The FDDI ring operates at a much higher rate than the Ethernet and may overwhelm the Ethernet interface with packets that need to be switched onto the Ethernet segment. If this happens in your internetwork, the same steps used in optimizing a serial interface, i.e., of disabling fast switching by use of the no ip route-cache command on the Ethernet interface, and adjusting buffers and hold queues, might help.
In most cases, drops on an Ethernet interface are rare. This is due in part to the way Ethernet operates. Any node at any time can transmit on the LAN cable as long as no other node is using the cable. Therefore packets are stored in a router for a relatively short period of time before being switched out an Ethernet interface and do not get held in buffers or hold queues. On an Ethernet interface, congestion normally displays itself through packet collisions after the packets have been transmitted out of an interface.
Collisions occur when two interfaces try to transmit a packet onto the network cable at the same time. A small number of collisions is expected in an Ethernet network, but the obvious question is what constitutes "a small number." That is different for each network; if you see a collision more than once every 3 to 5 seconds, you should investigate the source.
The source of collisions usually is the result of cabling that is too long, overly high utilization, or what is commonly referred to as a deaf node, one that has faulty circuitry that keeps it from hearing other nodes using the network cable.
Again, we have the question of what utilization is too high. Again the answer is that it is different for every network. If your network has a high percentage of broadcast packets, the utilization at which performance starts to degrade will be lower than for a network with few broadcasts. Typically, if the network utilization reaches 40 percent, you can expect problems. The way you detect utilization on a Cisco router is by the throughput on the interface. This statistic as reported by the 5-minute average is not particularly useful, as short peaks in utilization are not reported. If your router is reporting excessive collisions for any reason, you are better off using a LAN analyzer to determine if utilization is the cause of the problem.
Out-of-specification cabling causes more collisions because a node will listen to the network cable for a set period of time before determining that no other node is using it. The further away another node is, the greater the propagation delay of the signal from that node to the listening node. If a node is further away than the specification states, there will be more instances when a listening node determines that the network cable is free, when in fact the cable is carrying a packet that has not yet reached the listening node. Under such conditions, a collision will result when the listening node transmits a packet.
The only time you should have to look closely at the router Ethernet interface configuration is when the interface and line protocol report an up state and all physical connections to another node are good, but they still cannot communicate. The probable cause of this problem is that the two nodes are using incompatible frame types. The frame type is displayed on the output of the show interface ethernet 0 command. If you know the frame type used by the other node, it should be a simple mater to match them up.
If you need to communicate with two devices on the same network that use different frame types, you can use sub-interfaces on the router interface and assign different encapsulation frame types for each sub-interface. A sample configuration for assigning both the ethernet_ii and novell-ether (802.3) frame types to different sub-interfaces on the Ethernet 0 physical interface for a Novell network is given here.
Interface ethernet 0.1
ipx network 1234 encapsulation arpa
Interface ethernet 0.2
ipx network 5678 encapsulation novell-ether
As you see, each sub-interface is considered a different network, since it is not possible to assign different frame types to the same interface for the same network number.
Troubleshooting Token-Ring
Token-Ring and Ethernet both handle communication between computer nodes on a local area network. That is where the similarity between these two technologies ends, however. Token-Ring is a deterministic system, which means that its performance is always predictable (in theory) and that only one node on the network has the choice of using the network at any given time. This is achieved by having a special packet, called the token, that constantly circulates the ring. The only node that can transmit is the one with the token. This method eliminates the possibility of collisions, but does potentially waste transmission time by always asking all nodes if they have anything to send on the network, even if they have nothing to send.
Because Token-Ring interfaces have to hold packets for transmission until the router receives the token, you sometimes need to adjust hold queue sizes from the default to eliminate packet drops. Drops are reported in the show interface tokenring command, as illustrated in Fig. 8-11.
Router1>show int to0
Tokenring0 is up, line protocol is up
Hardware is TMS380, address is 0000.3Oe2.c44d
Internet address is 194.3.3.2 255.255.255.0
MTU 4464 bytes, BW 16000 Kbit, DLY 630 usec, rely, 255/255, load 1/255 Encapsulation SNAP, loopback not set, keepalive set (10 sec)
ARP type: SNAP, ARP timeout 4:00j0
Ring speed: 16Mbps
Single ring node, Transparent Bridge Capable
Group address: 0x00000000, Functional Address: 0x08800000
Ethernet Transit OUI: 0x0000F8
Last input0:00:00, output 0:00:00, output hang never
Last clearing of "show interface" counters never
Output queue 0/40, 0 drops: input queue 0/100, 0 drops
5 minute input rate 9000 bits/sec, 9 packets/sec
5 minute output rate 1000 bhs/sec, 3 packets/sec
7471015 packets input, 1593087959 bytes, 0 no buffer
Received 8888699 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
2902191 packets output, 253383249 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets, 0 restarts
0 output buffer failures, 0 output buffers swapped out
4 transitions
Figure 8-11: Output of the show interface tokenring 0 command
When a Token-Ring network is initialized, one node on the ring is elected the Active Monitor. This node is responsible for monitoring the passage of the token around the ring, deleting duplicate tokens if they occur, and generally making sure that everything runs smoothly. The process of electing an Active Monitor is transparent to network users and to network administrators. The show interface tokenring command will display the ring status as "initializing" if the router has just been connected to the ring. This is because the router interface will need to be added into the sequence of computers to receive the token. This process is referred to as the router interface "being inserted into the ring." If a hardware error has occurred, the show interface tokenring command will report an interface status of Reset.
The most common problems reported on token rings are incompatible ring speeds, ring beaconing conditions, source routing not enabled, or a large number of transitions. Ring speeds can be set either to 4 Mbps or to 16 Mbps with the commands shown as follows.
Router1(config)#interface tokenring 0
Router1(config-int)#ring-speed 4
The default is 16 Mbps, and if the interface is changed to 4 Mbps with the command above, the interface speed can be changed back again with the ring-speed 16 configuration command. The current ring speed setting is given in the screen display of the show interface tokenring command.
Ring beaconing occurs when a serious problem with a network component, such as the cabling, Multistation Access Unit (MSAU), or a node interface is detected. As a token gets passed around a ring, each node receives the token from its Nearest Active Upstream Neighbor (NAUN). Should a node not receive a valid token from its NAUN, the node will report a beacon condition to all other stations on the network. In theory, this should initiate an auto-reconfiguration process, whereby the suspect NAUN and node reporting the beacon are taken out of the ring and normal operation is restored for the rest of the ring. I have rarely seen this work in practice. If a serious problem exists and a node sends out beacon frames, the problem generally has to be fixed by manual intervention before the ring is operational again.
Source routing was discussed in detail in Chap. 5. The show interface tokenring command display tells you whether the interface is configured for source routing. The display will either indicate the interface is a single ring node or multi-ring node. Multi-ring nodes are enabled to collect and utilize RIF field information and therefore can participate in source routing. This line also tells you the type of bridging for which the interface is enabled. The interface whose show interface tokenring command is displayed in Fig. 8-11 is enabled only for transparent bridging and not for source route bridging. For the commands needed to enable source route bridging on an interface, refer to the section on Chap. 5 on source route bridging.
There are other commands that will give you information on Token-Ring packets, but little additional information is presented that will assist in the resolution of common problems. When working with Cisco technical support, you may be required to note the displays given by the show controllers tokenring command, which gives information on the hardware interface and summary packet statistics. The other command that sometimes is used is the debug tokenring command that identifies the MAC source and destination addresses for every packet that passes through the interface.

 


 
Books24x7.com, Inc © 2000 –  Feedback